Method and device to speed up face recognition

ABSTRACT

Method to customize an application associated with a television experience based on the recognition of users located in front of a display and in the field of view of a camera, comprising the following steps:an initialization step during which each user is enrolled in a database of a computer system and is defined by a profile referenced by a profile ID and comprising the user name, biometric data and additional personal data,a finding step during which a wide image, acquired by said camera is scanned to isolate at least one user&#39;s faces, to define a marking area surrounding it, to memorize the position of said marking areas,a matching step to extract the biometric data from said marking area, to match them with the biometric data of the profiles stored in the database, and to assign the detected profile ID with the marking area.While subsequent identification is requested by the application, the following steps are executedacquiring a wide image with the camera,extracting the areas of said image according to the marking areas,extracting for a particular marking area the biometric data of a face,starting the comparison of the extracted biometric data with the biometric data of the profile ID related to this marking area, and in absence of match, continuing with the other biometric data of the database until one profile is found,transmitting the found profile ID to the application.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.16/237,303, filed Dec. 31, 2018, which is a continuation of U.S. patentapplication Ser. No. 15/648,737 filed Jul. 13, 2017, which is acontinuation of U.S. patent application Ser. No. 13/992,914, filed Aug.26, 2013, which is a National Phase of PCT/EP2011/072174, filed on Dec.8, 2011, which claims priority to U.S. Provisional Patent ApplicationNo. 61/421,645, filed on Dec. 10, 2010 and European Patent ApplicationNo. 11154824.4, filed on Feb. 17, 2011, in the European Patent Office(EPO), the entire contents of each of which are incorporated byreference herein in their entirety.

INTRODUCTION

The present invention concerns the field of digital TV and in particularthe means to recognize views in front of a television.

BACKGROUND ART

The TV experience can be improved with new interactive features takinginto account the individuality of TV viewers: content recommendation,Electronic Program Guide with preferred channels, personal widget space,parental control . . . .

In this context, it becomes important to know who is actually watchingTV. To answer this question, it is foreseen to use biometric sensors torecognize householders who have originally been enrolled in the system.The benefit is twofold: some biometric technologies ensureeffortless/covert identification; in addition, they usually offer ameans to reliably authenticate users.

Face recognition is a very promising technology. However, a cameramounted on top of TV embraces a wide area encompassing multiple people.If a face recognition technology for video surveillance is run, severalproblems have to be solved:

-   -   The face recognition algorithm is computationally intensive and        the CPU load is proportional to the number of people in the        scene. For video-surveillance applications, the algorithm is        usually run on a high-end PC whereas the TV viewer        identification is supposed to run on limited CPU. If our goal is        to identify all the TV viewers to provide a personalized        experience for the group, the problem we have to solve is to        simplify the algorithm taking into account the specifics of the        scene in order to ease the integration in a CE device.    -   Though, it looks quite difficult to provide a personalized TV        experience for a group of people since it is tough to infer a        group's profile from multiple individual profiles. In addition,        not all the people in the scope of the camera are interested in        the TV experience.

Therefore, an alternative solution is to identify a unique TV viewer whowill act as a primary viewer in order to subsequently personalize the TVexperience for this person only. We will assume that this primary vieweris the one who holds the remote control. The problem we have to solve isto recognize the face of the person holding the remote control.

DISCLOSURE OF INVENTION

The aim of the invention is reached with a method to customize anapplication associated with a television experience based on therecognition of users located in front of a display and in the field ofview of a camera. This method comprises the following steps:

an initialization step during which each user is enrolled in a databaseof a computer system and is defined by a profile referenced by a profileID and comprising the following data:

-   -   a user name,    -   biometric data of said user acquired by said camera through a        face recognition process,    -   additional personal data of said user,        a finding step during which a wide image, acquired by said        camera and embracing a wide area to encompass all potential        users located in front of said display, is scanned in order to:    -   isolate at least one user's faces appearing on said wide image,    -   define a marking area surrounding each isolated face,    -   memorize the position of each of said marking areas in a        register that identify said marking areas,        a matching step during which each marking area is processed in        order to:    -   extract biometric data of the face appearing inside said marking        area,    -   match said extracted biometric data with the biometric data of        the profiles stored in the database, and assigning the detected        profile ID with the marking area,        while subsequent identification is requested by the application,        executing the following steps:    -   acquiring a wide image with the camera,    -   extracting from the wide image an active area according to the        marking area,    -   extracting for the active area the biometric data of a face,    -   starting the comparison of the extracted biometric data with the        biometric data of the profile ID related to this marking area,        and in absence of match, continuing with the other biometric        data of the database until one profile is found,    -   transmitting the found profile ID to the application.

Further steps could include that, in the case of no match duringsubsequent identifications, the found profile ID related to the markingarea is updated. Preferably, such an updating will be made in case thesame profile ID has been detected at several times.

Other optional steps and embodiments are disclosed in the followingdescription of the invention.

BRIEF DESCRIPTION OF THE FIGURES

The present invention will be better understood thanks to the attachedfigures in which:

FIG. 1 illustrates the position of a viewer

FIG. 2 illustrates the division of the scene into areas

FIG. 3 illustrates the wave length of the IR emitter

FIG. 4 illustrates the blue point of the remote control

DESCRIPTION OF THE INVENTION

According to the preferred embodiment of this invention, the methodfirstly aims to enroll each user in a database of a computer systemduring an initialization step in which each user is defined by a profilereferenced by a profile ID. Typically, the user's profile comprisesseveral data such as the user name, biometric data of the user andadditional personal data.

The biometric data of the enrolled users can be acquired by using facerecognition technologies.

People that want to be recognized by the computer system are enrolled bymeans of a snapshot of their face taken by a camera. For instance a mugshot is captured and the related biometric data are stored in a databasewithin the user's profile together with the identity of the person and,preferably with additional data specific to the user.

Once the enrolment is done, the method further comprises two main steps,namely a finding step and a matching step.

The finding step consists in scanning a wide image in search of faces.The image encompasses the front of the television and thus is able totake all persons currently watching TV. The image is processed toisolate faces and then, once a face is localized, a marking are (i.e. arectangle) containing the face is cropped and its position is memorizedin a register that identify the marking areas related to the faces ofthese persons.

The matching step compares the biometric data of the faces withcorresponding data recorded in the database. Once the biometric data ofa face recorded in the database is detected as being sufficiently closeto the biometric data of the detected face, the identity of the personis provided.

To this end, each marking area is processed in order to read thebiometric data appearing within the marking area. Then, these data arecompared with those of the profiles stored in the database and finallythe detected profile ID is assigned to the marking area so as toestablish a link between the marking area and the person related to thisarea.

The claimed solution is executed thanks to a device comprising a memory,in which the program containing the above method is stored, a processingunit that execute the program and carry out the acquisition step,comparison step, matching step and detection step, and a communicationchannel with an application in relation to a television experience. Thisdevice can be standalone, for example integrated in the camera or beingpart of a television or being implemented into a set-top-box.

One solution to reduce the CPU requirement when executing this method ona STB or a TV is obviously to reduce the frame rate i.e. treat 1 frameper second. However, it is likely that the frame is not usable for facerecognition because the TV viewer is moving and/or the head is not inupfront position. Instead of sub-sampling the video frames, othersolutions take benefit of the specifics of the scene.

Indeed, one can claim that people watching TV usually seat on sofa orchairs that are often put on a fixed position. If we introduce thisassumption in the present method, we can limit the scanning to a reducedportion of the scene which could be defined as follows:

At the very first scanning (initialization), the scanning is necessarilyexhaustive. Once a face is localized and a cropped rectangle containingthe face is extracted, a larger rectangle centred on the formerrectangle is marked in a fast scanning area bitmap so as to subsequentlyinstruct the finding step in nominal mode.

Therefore, in nominal mode, the scanning may be, for instance, limitedto this limited area. However, as we cannot exclude that somebodyappears in an unexpected area of the image or the furniture is moved,the exhaustive scanning will be preferably completed from time to time(e.g. 1 image every 10) in order to check that the system has not missedanybody. If so, the fast scanning area bitmap is updated with the newface position.

The following FIG. 1 illustrates the definition of the fast scanningarea; the dotted line rectangles represent the resulting area.

In addition, one can claim that people in a family usually sit at thesame position i.e. on the same piece of furniture. If we introduce thissecond assumption in the matching algorithm, we can improve therecognition speed since each cropped face provided by the finding stepwill be first matched against the most likely viewer, given the positionof the cropped face. If it happens that the computed distance betweenboth biometric data is too high, the algorithm further tries the otherrecords of the enrolment database.

To this end, the method of the present invention suggests at least onesubsequent identification of the persons located in the field of view ofthe camera. Such a subsequent identification may be requested by theapplication and the following steps could be executed:

-   -   acquiring a new wide image with the camera,    -   extracting from the wide image an active area according to the        marking area,    -   extracting for the active area the biometric data of a face,    -   starting the comparison of the extracted biometric data with the        biometric data of the profile ID related to this marking area,        and in absence of match, continuing with the other biometric        data of the database until one profile is found,    -   transmitting the found profile ID to the application.

Considering now the memorization of the likeliest identity, one simplesolution consists in logging the latest position (e.g. coordinates ofthe centre of the marking area) of each recognized person. Thus, at thenext time the matching algorithm is run, the detected face is firstmatched against the face of the person who sat at the nearest position.That can be made by updating the found profile ID related to the markingarea in case no match could be established during subsequentidentification of persons.

Another alternative could be to compute statistics rather than keepingthe last position. Indeed, if we look at the past positions of the facefor a given viewer, it is likely that a cloud of points will appear atsome area of the image whereas a few points will be isolated. If weoverlay a grid on the image, we can consider that the default positionof a TV viewer is the centre of the square which contains the highestamount of points.

Note that each point actually represents the average position of a TVviewer during a viewing session (a viewing session starts when a personhas been recognized in the scene for a certain time e.g. 1 minute and itends when a person has disappeared for this same duration).

FIG. 2 below illustrates the various positions of the face of a given TVviewer.

Then, at the next time the matching algorithm is run, the detected faceis first matched against the face of the person having the closestdefault position.

In an alternative solution, once a face is found, the centre of the faceis placed on the grid and the matching algorithm will first try to matchthe face against the face of the TV viewer who has spent the longesttime in the related square.

Thus, the steps relating to subsequent identification of persons couldinclude further conditions stating that, in case no match could beestablished during the comparison of the extracted biometric data withthe biometric data of the profile ID related to the current markingarea, then this profile ID must be updated in case several same profileshave been detected successively.

In a another embodiment of the invention, we will strive to recognizethe face of the primary TV viewer holding a remote control in order toprovide a TV experience for this person. As a consequence, the facerecognition is advantageously less computationally intensive compared toa multi-viewer scenario.

In the following, we will assume that TV device (Set Top Box, TV . . . )is controlled with a remote control.

The primary user is characterized by the fact he/she holds the remotecontrol to command the TV device and the remote control emits commandsignals which are usually Infra-Red light signals or Radio Frequencysignals.

In case the remote control is a RF remote control or in case thesensitivity of the camera in the IR frequencies is too weak to detectthe IR light, it may be necessary to add a dedicated beacon light to theremote control device

Therefore, the remote control can be used as a beacon light to localizethe primary user in the image or video captured by the camera when theuser presses buttons of the remote control.

In case the remote control is based on IR light, the wavelength of thisIR light is usually around 950 nm.

Standard cameras are usually equipped with an IR-cut filter mounted infront of the sensor in order to get rid of interferences due tosunlight, fluorescent bulbs . . . . If the IR filter is removed todetect the remote control signal, we can notice in the graph below (seeFIG. 3 ) that the intensity of the response spectrum is low at 950 nm.

Therefore, the impact of the remote control signal on the video may betoo weak to be detected. In addition, another drawback is that theelimination of the IR filter may lead to a significant loss of imagequality which is inconsistent with the face recognition requirements.

The proposed solution consists in using a remote control device whichemits a light marker in the visible spectrum for which the camera sensorhas a significant response. To avoid interference with ambient light andease the detection of this light marker, the light wavelength may be inthe blue range (around 465 nm) or near Ultra-Violet range (around 360nm) since natural or artificial light source contain little blue light.

According to a possible embodiment, the camera can comprise an IRdetector that is used to trigger the detection of the light marker.Since the remote will simultaneously emit an IR signal for thetelevision and the light marker, the camera can acquire an image onlywhen an IR signal is received, so that any light source having the samewave length as the light marker will not interfere with the lightmarker. In order to eliminate these interferences, it is possible toacquire an image after the end of the emission of the IR signal and thelight marker, so that remaining light source having the same wave lengthas the light marker can be localized and removed for the analysis of theposition of the light marker.

To this aim, the method of the present invention can also comprise thefollowing steps:

-   -   detecting the position of a signal emitted by the remote control        within the wide image acquired by the camera,    -   determining the distances between the signal's position and at        least one marking area position memorized in the register        identifying these marking areas,    -   identifying the profile ID related to the marking area which        corresponds to the shortest distance.

In the particular case where the process determines that two persons arelocated at the same distance from the remote control, or at very closedistances, a solution aiming to determine who is the primary user amongthese two persons could take into account an additional parameterspecified in the user's profiles, for instance a personal data informingthat the person is left-handed or right handed.

Incidentally, and not necessarily in view of any particular case, otherpersonal data could be taken into account in order to determine who isthe primary user among the persons located in front of the camera. Suchother personal data could be mother tongue of the user or his age withinthe framework of a parental control for example.

In addition, the assignment of the status of primary user could bedependent on a degree of priority recorded in the profile of the users.Therefore, a child could have a higher degree of priority than an adult,although the preceding steps of the process have assigned the positionof the remote control to this adult.

To this aim, the method of the present invention could further comprisethe steps of:

-   -   comparing at least one personal data common to the found profile        IDs, then    -   assigning a degree of priority to at least one of these profile        IDs on the basis of at least one predetermined criterion.

Typically, such a criterion could be based on the user's mother tongueor his age, such as a minimum age, or a combination of severaladditional personal data.

Alternatively, the remote control beacon light may be a secondadditional light which is dedicated to signal the position of the remotecontrol in the scene, in addition to the existing IR light signal. Incase the remote control communicates with the device using RF signals,this second option is necessarily used. This beacon light may be apermanent source or preferably a sporadic source which is emitted whensomeone presses a remote control button.

Since the TV device simultaneously receives a command from the IR or RFchannel, it can inform the camera that the beacon light is presentsomewhere in the scene.

From the image analysis standpoint, the localization of the remotecontrol light may be carried out by filtering the video baseband data toextract the Blue component which conveys most of the beacon signalemitted by the remote control. Then, once an intense blue light sourceis detected in the scene, the focus can be put on the correspondingimage area as depicted in the FIG. 4 below:

It should be noticed that the focus area is actually the one used byface recognition algorithms to find a face. In case the camera sensordoes not have a suitable resolution to carry out face recognition, itmay be possible to use a variable lens and to control that lens and thecamera itself with a motor to optically focus on the right area andproject the focus area on the whole sensor.

In case the distance between TV viewers is too short and the face finderdetects several faces in the focus area, a message may warn people thatthey are too close to identify the one who holds the remote control.

The invention claimed is:
 1. A method,. comprising: acquiring a firstimage, isolating a first face appearing in the first image, defining amarking area surrounding the isolated first face, extracting biometricdata of the first face appearing inside the marking area, associatingthe extracted biometric data with a profile ID, acquiring a second,determining a position of a second face in the second image surroundedby the defined marking area, extracting biometric data based of thesecond face appearing in the second image, based on determining that theextracted biometric data of the second face matches the biometric dataassociated with the profile ID, transmitting the profile ID to anapplication; based on determining that the extracted biometric data ofthe second face does not match the biometric data associated with theprofile ID, comparing the extracted biometric data of the second imagewith other biometric data.
 2. The method of claim 1, wherein the markingarea represents a portion of the first image.
 3. The method of claim 1,wherein, based on the determining that the extracted biometric data ofthe second iago face does not match with the first biometric dataassociated with the profile ID, the method further comprises: comparingthe extracted biometric data of the second face with other biometricdata, and identifying a different person matching the extractedbiometric data of the second face; and associating the identifieddifferent person with the marking area.
 4. The method of claim 1,further comprising: detecting a position of a signal emitted by a remotecontroller within the first image; determining the marking area ascorresponding to a shortest distance among distances between the signalposition and at least one marking area position; and identifying aperson associated with the biometric data of the first face based on themarking area.
 5. The method of claim 4, wherein the signal is anInfra-Red light signal.
 6. The method of claim 4, wherein the signal isa beacon signal for signaling a position of the remote controller. 7.The method of claim 6, wherein the beacon signal has a wavelength in ablue light frequency range.
 8. The method of claim 6, wherein the beaconsignal has a wavelength in a near Ultra-Violet light frequency range. 9.The method of claim 6, wherein the beacon signal is an intermittentsignal.
 10. The method of claim 1, further comprising: isolating one ormore additional user faces appearing in the first image, extractingbiometric data of each one of the one or more additional user faces,associating the extracted biometric data of each one of the one or moreadditional user faces with respective additional users, comparingpersonal data of an identified person and the additional users, andassigning a degree of priority to at least one of the identified personand the additional users based on at least one criterion.
 11. Anelectronic device, comprising: a display to display user relatedinformation of an application; processing circuitry configured toexecute computer-readable instructions to acquire the first image,isolate a first face appearing in the first image, define a marking areasurrounding the isolated first face, extract biometric data of the firstface appearing inside the marking area, associate the extractedbiometric data with a profile ID, acquire a second image, determine aposition of a second face in the second image surrounded by the definedmarking area, extract biometric data based of the second face appearingin the second image, based on determining that the extracted biometricdata of the second face matches the biometric data associated with theprofile ID, transmit the profile ID to the application; and based ondetermining that the extracted biometric data of the second face doesnot match the biometric data associated with the profile ID, compare theextracted biometric data of the second image with other biometric data.12. The device of claim 11, wherein the marking area represents aportion of the first image.
 13. The device of claim 11, wherein, basedon the determining that the extracted biometric data of the second facedoes not match with the biometric data associated with the profile ID,the processing circuitry is further configured to compare the extractedbiometric data of the second face with other biometric data, andidentify a different person matching the extracted biometric data of thesecond face; and associate the identified different person with themarking area.
 14. The device of claim 11, wherein the processingcircuitry is further configured to: detect a position of a signalemitted by a remote controller within the first image; determine themarking area as corresponding to a shortest distance among distancesbetween the signal position and at least one marking area position; andidentify a person associated with the extracted biometric data of thefirst face based on the marking area.
 15. The device of claim 11,wherein the processing circuitry is further configured to: isolate oneor more additional user faces appearing in the first image, extractbiometric data of each one of the one or more additional user faces,associate the extracted biometric data of each one of the one or moreadditional user faces with respective additional users, compare personaldata of an identified person and the additional users, and assign adegree of priority to at least one of the identified person and theadditional users based on at least one criterion.
 16. A non-transitorycomputer readable medium storing computer readable instructions, which,when executed by processing circuitry, cause the processing circuitryto: acquire a first image, isolate a first face appearing in the firstimage, define a marking area based on the first image surrounding theisolated first face, extract biometric data of the first face appearinginside the marking area, associate the extracted biometric data with aprofile ID, acquire a second image, determine a position of a secondface in the second image surrounded by the defined marking area, extractbiometric data based of the second face appearing in the second image,based on determining that the extracted biometric data of the secondface matches the biometric data associated with the profile ID, transmitthe profile ID to an application; and based on determining that theextracted biometric data of the second face does not match the biometricdata associated with the profile ID, compare the extracted biometricdata of the second image with other biometric data.
 17. Thenon-transitory computer readable medium of claim 16, wherein, based onthe determining that the extracted biometric data of the second facedoes not match with the biometric data associated with the profile ID,the processing circuitry is further caused to: compare the correlatedsccond extracted biometric data of the second face with other biometricdata, and identify a different person matching the extracted biometricdata of the second face; and associate the identified different personwith the marking area.
 18. The non-transitory computer readable mediumof claim 16, wherein the processing circuitry is further caused to:detect a position of a signal emitted by a remote controller within thefirst image; determine the marking area as corresponding to a shortestdistance among distances between the signal position and at least onemarking area position; and identify a person associated with theextracted biometric data of the first face based on the marking area.